{
  "cells": [
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/pinecone-io/examples/blob/master/learn/search/hybrid-search/fast-intro/pubmed-bm25.ipynb) [![Open nbviewer](https://raw.githubusercontent.com/pinecone-io/examples/master/assets/nbviewer-shield.svg)](https://nbviewer.org/github/pinecone-io/examples/blob/master/learn/search/hybrid-search/fast-intro/pubmed-bm25.ipynb)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4UAkYO8XUDxW"
      },
      "source": [
        "# Hybrid Search in Pinecone"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CrtPzpDfUN56"
      },
      "source": [
        "Pinecone now supports search across dense and sparse vectors with the introduction of the new hybrid vector index \u2014 a first-of-its-kind solution that lets engineers easily build keyword-aware semantic search into their applications. In this notebook, we will demonstrate how to use the new hybrid vector index.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ug-NsCoZViN8"
      },
      "source": [
        "## Install Dependencies"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NPPKJbKT_82b",
        "outputId": "da7ca7a9-fbae-40d7-d132-16e26baa8902"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting datasets\n",
            "  Downloading datasets-2.6.1-py3-none-any.whl (441 kB)\n",
            "\u001b[K     |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 441 kB 33.5 MB/s \n",
            "\u001b[?25hCollecting transformers\n",
            "  Downloading transformers-4.23.1-py3-none-any.whl (5.3 MB)\n",
            "\u001b[K     |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 5.3 MB 48.6 MB/s \n",
            "\u001b[?25hCollecting sentence-transformers\n",
            "  Downloading sentence-transformers-2.2.2.tar.gz (85 kB)\n",
            "\u001b[K     |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 85 kB 5.3 MB/s \n",
            "\u001b[?25hRequirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (2.23.0)\n",
            "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.7/dist-packages (from datasets) (1.21.6)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.7/dist-packages (from datasets) (21.3)\n",
            "Collecting multiprocess\n",
            "  Downloading multiprocess-0.70.14-py37-none-any.whl (115 kB)\n",
            "\u001b[K     |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 115 kB 69.3 MB/s \n",
            "\u001b[?25hRequirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/dist-packages (from datasets) (4.13.0)\n",
            "Requirement already satisfied: pyarrow>=6.0.0 in /usr/local/lib/python3.7/dist-packages (from datasets) (6.0.1)\n",
            "Requirement already satisfied: dill<0.3.6 in /usr/local/lib/python3.7/dist-packages (from datasets) (0.3.5.1)\n",
            "Collecting huggingface-hub<1.0.0,>=0.2.0\n",
            "  Downloading huggingface_hub-0.10.1-py3-none-any.whl (163 kB)\n",
            "\u001b[K     |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 163 kB 71.1 MB/s \n",
            "\u001b[?25hCollecting xxhash\n",
            "  Downloading xxhash-3.1.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (212 kB)\n",
            "\u001b[K     |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 212 kB 60.8 MB/s \n",
            "\u001b[?25hRequirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.7/dist-packages (from datasets) (6.0)\n",
            "Requirement already satisfied: fsspec[http]>=2021.11.1 in /usr/local/lib/python3.7/dist-packages (from datasets) (2022.10.0)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from datasets) (1.3.5)\n",
            "Requirement already satisfied: tqdm>=4.62.1 in /usr/local/lib/python3.7/dist-packages (from datasets) (4.64.1)\n",
            "Collecting responses<0.19\n",
            "  Downloading responses-0.18.0-py3-none-any.whl (38 kB)\n",
            "Requirement already satisfied: aiohttp in /usr/local/lib/python3.7/dist-packages (from datasets) (3.8.3)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests) (3.0.4)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests) (2.10)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests) (2022.9.24)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests) (1.24.3)\n",
            "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.7/dist-packages (from aiohttp->datasets) (1.2.0)\n",
            "Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.7/dist-packages (from aiohttp->datasets) (4.0.2)\n",
            "Requirement already satisfied: asynctest==0.13.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp->datasets) (0.13.0)\n",
            "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.7/dist-packages (from aiohttp->datasets) (6.0.2)\n",
            "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp->datasets) (1.8.1)\n",
            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp->datasets) (22.1.0)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4 in /usr/local/lib/python3.7/dist-packages (from aiohttp->datasets) (4.1.1)\n",
            "Requirement already satisfied: charset-normalizer<3.0,>=2.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp->datasets) (2.1.1)\n",
            "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.7/dist-packages (from aiohttp->datasets) (1.3.1)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from huggingface-hub<1.0.0,>=0.2.0->datasets) (3.8.0)\n",
            "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging->datasets) (3.0.9)\n",
            "Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1\n",
            "  Downloading urllib3-1.25.11-py2.py3-none-any.whl (127 kB)\n",
            "\u001b[K     |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 127 kB 73.6 MB/s \n",
            "\u001b[?25hCollecting tokenizers!=0.11.3,<0.14,>=0.11.1\n",
            "  Downloading tokenizers-0.13.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.6 MB)\n",
            "\u001b[K     |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 7.6 MB 53.3 MB/s \n",
            "\u001b[?25hRequirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.7/dist-packages (from transformers) (2022.6.2)\n",
            "Requirement already satisfied: torch>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from sentence-transformers) (1.12.1+cu113)\n",
            "Requirement already satisfied: torchvision in /usr/local/lib/python3.7/dist-packages (from sentence-transformers) (0.13.1+cu113)\n",
            "Requirement already satisfied: scikit-learn in /usr/local/lib/python3.7/dist-packages (from sentence-transformers) (1.0.2)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from sentence-transformers) (1.7.3)\n",
            "Requirement already satisfied: nltk in /usr/local/lib/python3.7/dist-packages (from sentence-transformers) (3.7)\n",
            "Collecting sentencepiece\n",
            "  Downloading sentencepiece-0.1.97-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
            "\u001b[K     |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 1.3 MB 58.0 MB/s \n",
            "\u001b[?25hRequirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->datasets) (3.9.0)\n",
            "Collecting multiprocess\n",
            "  Downloading multiprocess-0.70.13-py37-none-any.whl (115 kB)\n",
            "\u001b[K     |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 115 kB 72.4 MB/s \n",
            "\u001b[?25hRequirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from nltk->sentence-transformers) (7.1.2)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (from nltk->sentence-transformers) (1.2.0)\n",
            "Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.7/dist-packages (from pandas->datasets) (2022.5)\n",
            "Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.7/dist-packages (from pandas->datasets) (2.8.2)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.7.3->pandas->datasets) (1.15.0)\n",
            "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn->sentence-transformers) (3.1.0)\n",
            "Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /usr/local/lib/python3.7/dist-packages (from torchvision->sentence-transformers) (7.1.2)\n",
            "Building wheels for collected packages: sentence-transformers\n",
            "  Building wheel for sentence-transformers (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for sentence-transformers: filename=sentence_transformers-2.2.2-py3-none-any.whl size=125938 sha256=317cbf69abc201565efd5d868d8cabeffaa55139a826350233ad0a90f4ba8594\n",
            "  Stored in directory: /root/.cache/pip/wheels/bf/06/fb/d59c1e5bd1dac7f6cf61ec0036cc3a10ab8fecaa6b2c3d3ee9\n",
            "Successfully built sentence-transformers\n",
            "Installing collected packages: urllib3, tokenizers, huggingface-hub, xxhash, transformers, sentencepiece, responses, multiprocess, sentence-transformers, datasets\n",
            "  Attempting uninstall: urllib3\n",
            "    Found existing installation: urllib3 1.24.3\n",
            "    Uninstalling urllib3-1.24.3:\n",
            "      Successfully uninstalled urllib3-1.24.3\n",
            "Successfully installed datasets-2.6.1 huggingface-hub-0.10.1 multiprocess-0.70.13 responses-0.18.0 sentence-transformers-2.2.2 sentencepiece-0.1.97 tokenizers-0.13.1 transformers-4.23.1 urllib3-1.25.11 xxhash-3.1.0\n",
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/, https://download.pytorch.org/whl/cu116\n",
            "Requirement already satisfied: torch in /usr/local/lib/python3.7/dist-packages (1.12.1+cu113)\n",
            "Requirement already satisfied: torchvision in /usr/local/lib/python3.7/dist-packages (0.13.1+cu113)\n",
            "Requirement already satisfied: torchaudio in /usr/local/lib/python3.7/dist-packages (0.12.1+cu113)\n",
            "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from torch) (4.1.1)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from torchvision) (2.23.0)\n",
            "Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /usr/local/lib/python3.7/dist-packages (from torchvision) (7.1.2)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from torchvision) (1.21.6)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->torchvision) (2022.9.24)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->torchvision) (3.0.4)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->torchvision) (1.25.11)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->torchvision) (2.10)\n"
          ]
        }
      ],
      "source": [
        "!pip install -qU \\\n",
        "    pinecone-datasets==0.7.0 \\\n",
        "    pinecone-client==3.1.0 \\\n",
        "    pinecone-text \\\n",
        "    torch \\\n",
        "    datasets \\\n",
        "    transformers \\\n",
        "    sentence-transformers \\\n",
        "    requests"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Initializing the Index\n",
        "\n",
        "Now we need a place to store these embeddings and enable a efficient vector search through them all. To do that we use Pinecone, we can get a [free API key](https://app.pinecone.io/) and enter it below where we will initialize our connection to Pinecone and create a new index."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import os\n",
        "from pinecone import Pinecone\n",
        "\n",
        "# initialize connection to pinecone (get API key at app.pinecone.io)\n",
        "api_key = os.environ.get('PINECONE_API_KEY') or 'PINECONE_API_KEY'\n",
        "\n",
        "# configure client\n",
        "pc = Pinecone(api_key=api_key)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now we setup our index specification, this allows us to define the cloud provider and region where we want to deploy our index. You can find a list of all [available providers and regions here](https://docs.pinecone.io/docs/projects)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from pinecone import ServerlessSpec\n",
        "\n",
        "cloud = os.environ.get('PINECONE_CLOUD') or 'aws'\n",
        "region = os.environ.get('PINECONE_REGION') or 'us-east-1'\n",
        "\n",
        "spec = ServerlessSpec(cloud=cloud, region=region)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Create the index:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "index_name = \"hybrid-test\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import time\n",
        "\n",
        "# check if index already exists (it shouldn't if this is first time)\n",
        "if index_name not in pc.list_indexes().names():\n",
        "    # if does not exist, create index\n",
        "    pc.create_index(\n",
        "        index_name,\n",
        "        dimension=384,\n",
        "        metric='dotproduct',\n",
        "        spec=spec\n",
        "    )\n",
        "    # wait for index to be initialized\n",
        "    while not pc.describe_index(index_name).status['ready']:\n",
        "        time.sleep(1)\n",
        "\n",
        "# connect to index\n",
        "index = pc.Index(index_name)\n",
        "# view index stats\n",
        "index.describe_index_stats()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jfQ8_Uqf_f-8"
      },
      "source": [
        "## Load Dataset\n",
        "\n",
        "We will use a Question Answering dataset specific to the medical domain. This is to show how the hybrid index helps to rank search results better when domain specifc terms are used. We will load the dataset from Huggingface dataset hub as follows:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 397,
          "referenced_widgets": [
            "d8cd630c99d24c3b8d6a1ccdd83be2f6",
            "c5f2baa1c30e48e69f74db0d9b22e22e",
            "3bb0f27d48a84de6af93a766a0e743a8",
            "76cf09f2eb1f432d960eed529d27b8ad",
            "6323d30c69874bac8c8639285bca04ad",
            "f548288aa2364f5c84c565454a4ff3e9",
            "7f4aa16d01b140aa8237583cb50be477",
            "b0a50f4d53fe4dc3bce74f374b293089",
            "caae57d42c524aa6ab3a4b8792bca121",
            "838f5bad749c41d5a3cba0432ff819ac",
            "b2a39e3b2fab475ca4eb9ccab7a42fd6",
            "663cfdd142d94db2931337445815b9bb",
            "341c91456fc34e949422f4fa3dbe1bc4",
            "0dcfbaee958f4f6b9033b7d9c03b1e84",
            "fcbb67302249467d8b21c455a4e7de32",
            "0ed7e07e30954bab9a6a13fee0983fc0",
            "32071627003645faae484baf075b5475",
            "5be476729c0c435e8cf0bf8f652c9c1f",
            "e926ee13e5de40d585935407761d6aee",
            "f5a3cff4d9594eb1b0759f7d97dd7d2f",
            "df094d7acec04c12aa4e5218307d69ce",
            "1a550058b63b4cae8ebaf8f5c3ab8403",
            "daff9413391a4cdc89edf19b30b14095",
            "71f8c2f7fa624eea90adc6afd601a0a7",
            "caf33879e0ca44ef9dc0d49ee6c00844",
            "11a4775bf175454f8d83bb01692e40f1",
            "5b3243f5a84a4a81abdc44706d6ab7be",
            "7b599fea323f4fb89f01728a6d53da74",
            "90f2a63370664c369198d9c446940e95",
            "b8a633675eeb4d559bc27d50a7302702",
            "c66a3218d063459fb143c47430f5e415",
            "21f01e4403444c95a25ecc3549758454",
            "24e28a58d923413db62629c222deb2bb",
            "478beaf4401d4f458dabd2e380cbfb65",
            "1a28425eb4e5485090459cc5ea12d70a",
            "2e3fc18deff64bb59555bb0f78add869",
            "25fa1a8a6dbc4c24b3a1f964496eb1e0",
            "9b12b18895f642efa232ef5b2d16214a",
            "a35fcf628a8d400b99edf891d5f07f37",
            "9537e8f4947e46038f52f121be88214f",
            "6204b080baaf470d90b5587980cf9e01",
            "c43197a4339546be994ea8e0fd2cf8b8",
            "66b293bdd8744eca913b612a23197e45",
            "24a36b033c584c589fa4100cf059af45",
            "a006439efca14035a5b5f4cccec9789f",
            "99e9be3e726943aca4082cbe66b7ef86",
            "7638233732f84c4fbc33e7535b5e8eee",
            "2dc5cdda50064864ab5607188c589ad1",
            "14df714dbf4e4c0f9b71ee64428034c9",
            "840eeee94d924feb9489fbe80f4fdcbe",
            "9d2457cb45954453ae73e89eba0fd533",
            "5a411d4f15974e258a3bbfe8ec562bdc",
            "4cff7d8ca00a4fdeb74b0b1285cc7660",
            "bc2b8343bb1c44c3ba35409c5d06cec6",
            "f37811fb571048939b39495c57165c78",
            "268e5bafd6574953a7e9382cd18737e0",
            "8bb75c00ba6c40d9b60840271c045c77",
            "d5ff3fff688b49d5afb66fee0c174f05",
            "3ec837d5f8e34a1d8024000d8639d8ce",
            "c92c06af936e425ca16ac7791430597a",
            "9bcd65d818624803a9cd34fbff55581e",
            "f064be9b7dd342e681ad467d172d1b1f",
            "21fdbaa5c31e4408ad34e3dcd5b64e0b",
            "a10e35d4154c4032b434270d6bbd5faf",
            "60dc89c38dec401ba52f3c2a9ea989fd",
            "b27de6cbef3d478d8a62b0a17f0d73b6",
            "d669e28936f145a49de0488c3172e0cd",
            "2d6c1173cf0846a89f69b647b4c702a3",
            "3accff0bea53493c9da7fabc03830aaa",
            "be249af036d4404a8fc95dc0dbd2a77d",
            "76faa9f70932481b80b510b8d18cf6c2",
            "09a9052b9f864051924792b62250d4be",
            "746290d608264027a67364ae828fba94",
            "c6f419f294f14a1aa9236ac18108cd45",
            "6a43e09e59a44a4cabc5680a69e7282b",
            "c90939587f5a4d1e84dee99d8b9434ca",
            "b565f4fe317e4c93b68609d0ae6d8f84",
            "83ca2d4cdbb540b88a814448385ca216",
            "56ce912f087a4d498373a91251911a52",
            "1b444cfb61bc4be390bef192c1e3c1d1",
            "db5a8dd673d54cf88cce2aecf741276e",
            "092b5e61ec594c32a3c4256cdf6d1517",
            "adb951e291504ad0a2367b3e2e2a917c",
            "f3babc6d71d144b097a17e3aa4be32cf",
            "3cdd2c19c56a4b16a7555404ea8f9365",
            "069ec320293543e4a466bed22783e133",
            "9cdb37d0a5aa40b68795dd1425685ecb",
            "fbe2d5d4b89840a8b7066707eb8bc7b3",
            "c56227dad1f94bf2ae2f10bdd142e2ff",
            "5b8ccca8a8e242629d8a017de75354b6",
            "af24bb071530406ca153a234fc71cd4f",
            "2be9818c9f6748ecb42f02418591ce58",
            "0e84432b69c74deaab852ef4342b8266",
            "b0c6b87b8ff54a2d8e32fff13da0ae86",
            "2ebf1d852f554c8a95fa04ae15b468f7",
            "9aa19c150e6c4902b65811368e1f7e74",
            "389ef156c131442aa19d735d1f6f2f83",
            "7f19ea49a1e44cbfacfa216a2e37548d",
            "ec50880ff7f2477481c3bbedf075bcb5"
          ]
        },
        "id": "RaaD0YT__f-8",
        "outputId": "bc44cca9-d8fb-47da-d299-a5d8b865c57e"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "d8cd630c99d24c3b8d6a1ccdd83be2f6",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading builder script:   0%|          | 0.00/11.1k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "663cfdd142d94db2931337445815b9bb",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading metadata:   0%|          | 0.00/12.7k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "daff9413391a4cdc89edf19b30b14095",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading readme:   0%|          | 0.00/3.09k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Downloading and preparing dataset pubmed_qa/pqa_labeled (download: 656.02 MiB, generated: 1.99 MiB, post-processed: Unknown size, total: 658.01 MiB) to /root/.cache/huggingface/datasets/pubmed_qa/pqa_labeled/1.0.0/dd4c39f031a958c7e782595fa4dd1b1330484e8bbadd4d9212e5046f27e68924...\n"
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "478beaf4401d4f458dabd2e380cbfb65",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading data files:   0%|          | 0/3 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "a006439efca14035a5b5f4cccec9789f",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading data:   0%|          | 0.00/709k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "268e5bafd6574953a7e9382cd18737e0",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading data:   0%|          | 0.00/152M [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "d669e28936f145a49de0488c3172e0cd",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading data:   0%|          | 0.00/533M [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "83ca2d4cdbb540b88a814448385ca216",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Extracting data files:   0%|          | 0/3 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "c56227dad1f94bf2ae2f10bdd142e2ff",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Generating train split:   0%|          | 0/1000 [00:00<?, ? examples/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Dataset pubmed_qa downloaded and prepared to /root/.cache/huggingface/datasets/pubmed_qa/pqa_labeled/1.0.0/dd4c39f031a958c7e782595fa4dd1b1330484e8bbadd4d9212e5046f27e68924. Subsequent calls will reuse this data.\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "Dataset({\n",
              "    features: ['pubid', 'question', 'context', 'long_answer', 'final_decision'],\n",
              "    num_rows: 1000\n",
              "})"
            ]
          },
          "execution_count": 10,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "from datasets import load_dataset\n",
        "\n",
        "# load the medical dataset from huggingfac dataset hub\n",
        "pubmed = load_dataset('pubmed_qa', 'pqa_labeled', split='train')\n",
        "pubmed"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0ZRSFjLW_f-8"
      },
      "source": [
        "Let's look at a set of contexts (which we will be storing)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JPJxRzLRzF4r",
        "outputId": "d8bac674-0d28-441a-a35b-578e5371310e"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Programmed cell death (PCD) is the regulated death of cells within an organism. The lace plant (Aponogeton madagascariensis) produces perforations in its leaves through PCD. The leaves of the plant consist of a latticework of longitudinal and transverse veins enclosing areoles. PCD occurs in the cel...\n",
            "Assessment of visual acuity depends on the optotypes used for measurement. The ability to recognize different optotypes differs even if their critical details appear under the same visual angle. Since optotypes are evaluated on individuals with good visual acuity and without eye disorders, differenc...\n"
          ]
        }
      ],
      "source": [
        "contexts = []\n",
        "# loop through the context passages\n",
        "for record in pubmed['context']:\n",
        "    # join context passages for each question and append to contexts list\n",
        "    contexts.append('\\n'.join(record['contexts']))\n",
        "# view some of the contexts\n",
        "for context in contexts[:2]:\n",
        "    print(f\"{context[:300]}...\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PyuqZHg_j22O"
      },
      "source": [
        "## Sparse Vectors"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pj_-5m9p_f-9"
      },
      "source": [
        "We will use the BERT tokenizer to create sparse vectors, i.e. we will take the token IDs."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 145,
          "referenced_widgets": [
            "d53abbf94f4940f6a5f8fee19287fcda",
            "4002881c1c7f4f15a16a26a49dcc8215",
            "f22e1a1391244f0c9ca035a9afa6a759",
            "37b48fe5f5a4434b989ade8ae01bfae6",
            "820dc52d19004802812f042529f1ab63",
            "d2e5c122610f4c26a2459208dc3a7e27",
            "91e1e2a2a6604412afe93a54d9464520",
            "6d37cb5c144846cf96f3ec8f2d317ec2",
            "2cee7bd96f984af9ae830d33569c2b75",
            "36da60255be6456d82e05c55408549c0",
            "9d94f23fc59945a79896415ca5dc7200",
            "9bf28f6509f84a7ea0395e25c9464b1d",
            "572ae4cb995e4eb889c97402436c4a14",
            "ff5f803b97234fc79f34809b616d4ba2",
            "a4f4c5dd1175485e91f4fa5a9e536d2c",
            "db32c27ce80d4de39e7a897613ab92e6",
            "b3eeda2cbddc4cd1915adee69e3db77f",
            "6f5c9b4180054fcebdf74a5c9f4e1328",
            "65f328726dc6492fb29df53da6ab02a6",
            "95305e6dcb544ceba78af0ee55688852",
            "4720331260eb4905b4bd45bf9aff4be4",
            "7de95b090f9f4178985feee7f35edb28",
            "260a2c3a78b3441e820522f4a4c04da8",
            "c2249564ce77497da3cfb1d9f42d7dcf",
            "60bc87fc20304983a3e574f8a36eaaed",
            "b094f99816004968b2273029a0d19cb7",
            "5c635821a85042d183eca9d842ef08d9",
            "55e240e2cdad4ad6a750d92b89ae2149",
            "48bd9d63184c4e228ac42c7337f37de2",
            "a375ae1e16b64390950ef76791486562",
            "e0c1273ac30849839e1a8fd4f6cdbf39",
            "908646d92b3e4e87ade563902f44496c",
            "3cec1dbc873b4483a231c7fbe50d60f0",
            "d4863fddae334c9f94030bacc37750d4",
            "06f81cf42e194e88b6f34ee490a89aff",
            "bf9164e0b39440359336c3fe8927be74",
            "5e30ff2ce7c84738b8021fbb48321870",
            "d5d09b5d397a4c238b29a7520e3ef26f",
            "4b87dcdab5aa4f248285975e530c303d",
            "b62253243f0246cfb3a82e4999bd7801",
            "ed1870107bb4437389775e83f29704a3",
            "5f8b8766c5fd41a7bbdfb8915dcb20cd",
            "ce1198d5237f4887be6066dd5f4ad73e",
            "ca931bc257a54695a8494968cfa3c775"
          ]
        },
        "id": "Zka805CX_f-9",
        "outputId": "7dd3dd50-5f97-4056-b77d-b5bf9c26377e"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "d53abbf94f4940f6a5f8fee19287fcda",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/28.0 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "9bf28f6509f84a7ea0395e25c9464b1d",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/232k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "260a2c3a78b3441e820522f4a4c04da8",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/466k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "d4863fddae334c9f94030bacc37750d4",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/570 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "from transformers import BertTokenizerFast\n",
        "\n",
        "# load bert tokenizer from huggingface\n",
        "tokenizer = BertTokenizerFast.from_pretrained(\n",
        "    'bert-base-uncased'\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rwCSrQR4kYOT"
      },
      "source": [
        "Let's create a sparse vector for a context passage."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "U8DMRNNF_f-9",
        "outputId": "f4a866ac-4101-43e5-f165-de344be73aba"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "dict_keys(['input_ids', 'token_type_ids', 'attention_mask'])"
            ]
          },
          "execution_count": 13,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# tokenize the context passage\n",
        "inputs = tokenizer(\n",
        "    contexts[0], padding=True, truncation=True,\n",
        "    max_length=512\n",
        ")\n",
        "inputs.keys()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DLsbyFfs_f-9",
        "outputId": "6bf1cb49-7036-4dde-a585-eb66b853a84e"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[101,\n",
              " 16984,\n",
              " 3526,\n",
              " 2331,\n",
              " 1006,\n",
              " 7473,\n",
              " 2094,\n",
              " 1007,\n",
              " 2003,\n",
              " 1996,\n",
              " 12222,\n",
              " 2331,\n",
              " 1997,\n",
              " 4442,\n",
              " 2306,\n",
              " 2019,\n",
              " 15923,\n",
              " 1012,\n",
              " 1996,\n",
              " 12922,\n",
              " 3269,\n",
              " 1006,\n",
              " 9706,\n",
              " 17175,\n",
              " 18150,\n",
              " 2239,\n",
              " 11934,\n",
              " 27806,\n",
              " 1007,\n",
              " 7137,\n",
              " 2566,\n",
              " 29278,\n",
              " 10708,\n",
              " 1999,\n",
              " 2049,\n",
              " 3727,\n",
              " 2083,\n",
              " 7473,\n",
              " 2094,\n",
              " 1012,\n",
              " 1996,\n",
              " 3727,\n",
              " 1997,\n",
              " 1996,\n",
              " 3269,\n",
              " 8676,\n",
              " 1997,\n",
              " 1037,\n",
              " 17779,\n",
              " 6198,\n",
              " 1997,\n",
              " 20134,\n",
              " 1998,\n",
              " 18323,\n",
              " 9607,\n",
              " 4372,\n",
              " 20464,\n",
              " 18606,\n",
              " 2024,\n",
              " 29111,\n",
              " 1012,\n",
              " 7473,\n",
              " 2094,\n",
              " 5158,\n",
              " 1999,\n",
              " 1996,\n",
              " 4442,\n",
              " 2012,\n",
              " 1996,\n",
              " 2415,\n",
              " 1997,\n",
              " 2122,\n",
              " 2024,\n",
              " 29111,\n",
              " 1998,\n",
              " 22901,\n",
              " 15436,\n",
              " 2015,\n",
              " 1010,\n",
              " 7458,\n",
              " 3155,\n",
              " 2274,\n",
              " 4442,\n",
              " 2013,\n",
              " 1996,\n",
              " 12436,\n",
              " 28817,\n",
              " 20051,\n",
              " 5397,\n",
              " 1012,\n",
              " 1996,\n",
              " 2535,\n",
              " 1997,\n",
              " 10210,\n",
              " 11663,\n",
              " 15422,\n",
              " 4360,\n",
              " 2076,\n",
              " 7473,\n",
              " 2094,\n",
              " 2038,\n",
              " 2042,\n",
              " 3858,\n",
              " 1999,\n",
              " 4176,\n",
              " 1025,\n",
              " 2174,\n",
              " 1010,\n",
              " 2009,\n",
              " 2038,\n",
              " 2042,\n",
              " 2625,\n",
              " 3273,\n",
              " 2076,\n",
              " 7473,\n",
              " 2094,\n",
              " 1999,\n",
              " 4264,\n",
              " 1012,\n",
              " 1996,\n",
              " 2206,\n",
              " 3259,\n",
              " 3449,\n",
              " 14194,\n",
              " 8524,\n",
              " 4570,\n",
              " 1996,\n",
              " 2535,\n",
              " 1997,\n",
              " 23079,\n",
              " 10949,\n",
              " 2076,\n",
              " 13908,\n",
              " 2135,\n",
              " 12222,\n",
              " 7473,\n",
              " 2094,\n",
              " 1999,\n",
              " 24269,\n",
              " 1999,\n",
              " 1037,\n",
              " 1012,\n",
              " 11934,\n",
              " 27806,\n",
              " 1012,\n",
              " 1037,\n",
              " 2309,\n",
              " 2024,\n",
              " 9890,\n",
              " 2306,\n",
              " 1037,\n",
              " 3332,\n",
              " 2754,\n",
              " 7053,\n",
              " 1006,\n",
              " 7473,\n",
              " 2094,\n",
              " 2003,\n",
              " 10066,\n",
              " 1007,\n",
              " 2001,\n",
              " 4055,\n",
              " 2046,\n",
              " 2093,\n",
              " 2752,\n",
              " 2241,\n",
              " 2006,\n",
              " 1996,\n",
              " 14967,\n",
              " 1997,\n",
              " 7473,\n",
              " 2094,\n",
              " 1025,\n",
              " 4442,\n",
              " 2008,\n",
              " 2097,\n",
              " 2025,\n",
              " 13595,\n",
              " 7473,\n",
              " 2094,\n",
              " 1006,\n",
              " 27937,\n",
              " 19797,\n",
              " 1007,\n",
              " 1010,\n",
              " 4442,\n",
              " 1999,\n",
              " 2220,\n",
              " 5711,\n",
              " 1997,\n",
              " 7473,\n",
              " 2094,\n",
              " 1006,\n",
              " 4958,\n",
              " 19797,\n",
              " 1007,\n",
              " 1010,\n",
              " 1998,\n",
              " 4442,\n",
              " 1999,\n",
              " 2397,\n",
              " 5711,\n",
              " 1997,\n",
              " 7473,\n",
              " 2094,\n",
              " 1006,\n",
              " 6948,\n",
              " 19797,\n",
              " 1007,\n",
              " 1012,\n",
              " 3332,\n",
              " 2754,\n",
              " 3727,\n",
              " 2020,\n",
              " 9702,\n",
              " 2007,\n",
              " 1996,\n",
              " 23079,\n",
              " 18554,\n",
              " 10210,\n",
              " 4140,\n",
              " 22648,\n",
              " 5484,\n",
              " 2417,\n",
              " 4642,\n",
              " 2595,\n",
              " 7352,\n",
              " 1998,\n",
              " 8920,\n",
              " 1012,\n",
              " 23079,\n",
              " 10949,\n",
              " 2020,\n",
              " 3972,\n",
              " 3170,\n",
              " 4383,\n",
              " 2046,\n",
              " 2176,\n",
              " 7236,\n",
              " 1006,\n",
              " 23290,\n",
              " 1011,\n",
              " 1049,\n",
              " 2549,\n",
              " 1007,\n",
              " 2241,\n",
              " 2006,\n",
              " 6459,\n",
              " 2164,\n",
              " 4353,\n",
              " 1010,\n",
              " 9587,\n",
              " 18724,\n",
              " 1010,\n",
              " 1998,\n",
              " 10804,\n",
              " 4022,\n",
              " 1006,\n",
              " 1158,\n",
              " 29738,\n",
              " 2213,\n",
              " 1007,\n",
              " 1012,\n",
              " 1037,\n",
              " 8694,\n",
              " 2140,\n",
              " 4632,\n",
              " 4710,\n",
              " 3662,\n",
              " 26872,\n",
              " 1050,\n",
              " 28911,\n",
              " 1999,\n",
              " 1037,\n",
              " 17978,\n",
              " 2058,\n",
              " 2122,\n",
              " 23079,\n",
              " 5711,\n",
              " 1012,\n",
              " 10381,\n",
              " 10626,\n",
              " 7361,\n",
              " 8523,\n",
              " 3215,\n",
              " 1998,\n",
              " 9099,\n",
              " 24887,\n",
              " 19098,\n",
              " 8017,\n",
              " 14119,\n",
              " 2020,\n",
              " 2036,\n",
              " 8920,\n",
              " 2478,\n",
              " 2444,\n",
              " 3526,\n",
              " 12126,\n",
              " 1012,\n",
              " 1996,\n",
              " 2825,\n",
              " 5197,\n",
              " 1997,\n",
              " 23079,\n",
              " 2566,\n",
              " 4168,\n",
              " 8010,\n",
              " 6653,\n",
              " 18499,\n",
              " 2063,\n",
              " 1006,\n",
              " 13866,\n",
              " 2361,\n",
              " 1007,\n",
              " 4195,\n",
              " 2076,\n",
              " 7473,\n",
              " 2094,\n",
              " 2001,\n",
              " 17351,\n",
              " 8920,\n",
              " 3081,\n",
              " 1999,\n",
              " 24269,\n",
              " 22330,\n",
              " 20464,\n",
              " 2891,\n",
              " 17822,\n",
              " 3170,\n",
              " 1037,\n",
              " 1006,\n",
              " 27804,\n",
              " 1007,\n",
              " 3949,\n",
              " 1012,\n",
              " 2023,\n",
              " 3949,\n",
              " 4504,\n",
              " 1999,\n",
              " 12922,\n",
              " 3269,\n",
              " 3727,\n",
              " 2007,\n",
              " 1037,\n",
              " 6022,\n",
              " 2896,\n",
              " 2193,\n",
              " 1997,\n",
              " 2566,\n",
              " 29278,\n",
              " 10708,\n",
              " 4102,\n",
              " 2000,\n",
              " 7711,\n",
              " 1010,\n",
              " 1998,\n",
              " 2008,\n",
              " 6913,\n",
              " 23079,\n",
              " 10949,\n",
              " 2714,\n",
              " 2000,\n",
              " 2008,\n",
              " 1997,\n",
              " 2512,\n",
              " 1011,\n",
              " 7473,\n",
              " 2094,\n",
              " 4442,\n",
              " 1012,\n",
              " 102]"
            ]
          },
          "execution_count": 14,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# extract the input ids\n",
        "input_ids = inputs['input_ids']\n",
        "input_ids"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PBNGXPMH_f--"
      },
      "source": [
        "We need to convert this into a dictionary of key to frequency values."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tDfAIC9vleBd",
        "outputId": "f7777643-58d1-48d0-c90e-c30a91d0b210"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{101: 1,\n",
              " 16984: 1,\n",
              " 3526: 2,\n",
              " 2331: 2,\n",
              " 1006: 10,\n",
              " 7473: 13,\n",
              " 2094: 13,\n",
              " 1007: 10,\n",
              " 2003: 2,\n",
              " 1996: 13,\n",
              " 12222: 2,\n",
              " 1997: 13,\n",
              " 4442: 7,\n",
              " 2306: 2,\n",
              " 2019: 1,\n",
              " 15923: 1,\n",
              " 1012: 14,\n",
              " 12922: 2,\n",
              " 3269: 3,\n",
              " 9706: 1,\n",
              " 17175: 1,\n",
              " 18150: 1,\n",
              " 2239: 1,\n",
              " 11934: 2,\n",
              " 27806: 2,\n",
              " 7137: 1,\n",
              " 2566: 3,\n",
              " 29278: 2,\n",
              " 10708: 2,\n",
              " 1999: 11,\n",
              " 2049: 1,\n",
              " 3727: 4,\n",
              " 2083: 1,\n",
              " 8676: 1,\n",
              " 1037: 8,\n",
              " 17779: 1,\n",
              " 6198: 1,\n",
              " 20134: 1,\n",
              " 1998: 7,\n",
              " 18323: 1,\n",
              " 9607: 1,\n",
              " 4372: 1,\n",
              " 20464: 2,\n",
              " 18606: 1,\n",
              " 2024: 3,\n",
              " 29111: 2,\n",
              " 5158: 1,\n",
              " 2012: 1,\n",
              " 2415: 1,\n",
              " 2122: 2,\n",
              " 22901: 1,\n",
              " 15436: 1,\n",
              " 2015: 1,\n",
              " 1010: 7,\n",
              " 7458: 1,\n",
              " 3155: 1,\n",
              " 2274: 1,\n",
              " 2013: 1,\n",
              " 12436: 1,\n",
              " 28817: 1,\n",
              " 20051: 1,\n",
              " 5397: 1,\n",
              " 2535: 2,\n",
              " 10210: 2,\n",
              " 11663: 1,\n",
              " 15422: 1,\n",
              " 4360: 1,\n",
              " 2076: 4,\n",
              " 2038: 2,\n",
              " 2042: 2,\n",
              " 3858: 1,\n",
              " 4176: 1,\n",
              " 1025: 2,\n",
              " 2174: 1,\n",
              " 2009: 1,\n",
              " 2625: 1,\n",
              " 3273: 1,\n",
              " 4264: 1,\n",
              " 2206: 1,\n",
              " 3259: 1,\n",
              " 3449: 1,\n",
              " 14194: 1,\n",
              " 8524: 1,\n",
              " 4570: 1,\n",
              " 23079: 6,\n",
              " 10949: 3,\n",
              " 13908: 1,\n",
              " 2135: 1,\n",
              " 24269: 2,\n",
              " 2309: 1,\n",
              " 9890: 1,\n",
              " 3332: 2,\n",
              " 2754: 2,\n",
              " 7053: 1,\n",
              " 10066: 1,\n",
              " 2001: 2,\n",
              " 4055: 1,\n",
              " 2046: 2,\n",
              " 2093: 1,\n",
              " 2752: 1,\n",
              " 2241: 2,\n",
              " 2006: 2,\n",
              " 14967: 1,\n",
              " 2008: 3,\n",
              " 2097: 1,\n",
              " 2025: 1,\n",
              " 13595: 1,\n",
              " 27937: 1,\n",
              " 19797: 3,\n",
              " 2220: 1,\n",
              " 5711: 3,\n",
              " 4958: 1,\n",
              " 2397: 1,\n",
              " 6948: 1,\n",
              " 2020: 3,\n",
              " 9702: 1,\n",
              " 2007: 2,\n",
              " 18554: 1,\n",
              " 4140: 1,\n",
              " 22648: 1,\n",
              " 5484: 1,\n",
              " 2417: 1,\n",
              " 4642: 1,\n",
              " 2595: 1,\n",
              " 7352: 1,\n",
              " 8920: 3,\n",
              " 3972: 1,\n",
              " 3170: 2,\n",
              " 4383: 1,\n",
              " 2176: 1,\n",
              " 7236: 1,\n",
              " 23290: 1,\n",
              " 1011: 2,\n",
              " 1049: 1,\n",
              " 2549: 1,\n",
              " 6459: 1,\n",
              " 2164: 1,\n",
              " 4353: 1,\n",
              " 9587: 1,\n",
              " 18724: 1,\n",
              " 10804: 1,\n",
              " 4022: 1,\n",
              " 1158: 1,\n",
              " 29738: 1,\n",
              " 2213: 1,\n",
              " 8694: 1,\n",
              " 2140: 1,\n",
              " 4632: 1,\n",
              " 4710: 1,\n",
              " 3662: 1,\n",
              " 26872: 1,\n",
              " 1050: 1,\n",
              " 28911: 1,\n",
              " 17978: 1,\n",
              " 2058: 1,\n",
              " 10381: 1,\n",
              " 10626: 1,\n",
              " 7361: 1,\n",
              " 8523: 1,\n",
              " 3215: 1,\n",
              " 9099: 1,\n",
              " 24887: 1,\n",
              " 19098: 1,\n",
              " 8017: 1,\n",
              " 14119: 1,\n",
              " 2036: 1,\n",
              " 2478: 1,\n",
              " 2444: 1,\n",
              " 12126: 1,\n",
              " 2825: 1,\n",
              " 5197: 1,\n",
              " 4168: 1,\n",
              " 8010: 1,\n",
              " 6653: 1,\n",
              " 18499: 1,\n",
              " 2063: 1,\n",
              " 13866: 1,\n",
              " 2361: 1,\n",
              " 4195: 1,\n",
              " 17351: 1,\n",
              " 3081: 1,\n",
              " 22330: 1,\n",
              " 2891: 1,\n",
              " 17822: 1,\n",
              " 27804: 1,\n",
              " 3949: 2,\n",
              " 2023: 1,\n",
              " 4504: 1,\n",
              " 6022: 1,\n",
              " 2896: 1,\n",
              " 2193: 1,\n",
              " 4102: 1,\n",
              " 2000: 2,\n",
              " 7711: 1,\n",
              " 6913: 1,\n",
              " 2714: 1,\n",
              " 2512: 1,\n",
              " 102: 1}"
            ]
          },
          "execution_count": 15,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "from collections import Counter\n",
        "\n",
        "# convert the input_ids list to a dictionary of key to frequency values\n",
        "sparse_vec = dict(Counter(input_ids))\n",
        "sparse_vec"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aOjHydCsl2OZ"
      },
      "source": [
        "Let's write a function to do this in batches. Notice that we are removing some keys from the dictionary. These are special tokens from the tokenizer which we do not really need when creating sparse vectors."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "id": "azF1uiGQln86"
      },
      "outputs": [],
      "source": [
        "def build_dict(input_batch):\n",
        "  # store a batch of sparse embeddings\n",
        "    sparse_emb = []\n",
        "    # iterate through input batch\n",
        "    for token_ids in input_batch:\n",
        "        # convert the input_ids list to a dictionary of key to frequency values\n",
        "        d = dict(Counter(token_ids))\n",
        "        # remove special tokens and append sparse vectors to sparse_emb list\n",
        "        sparse_emb.append({key: d[key] for key in d if key not in [101, 102, 103, 0]})\n",
        "    # return sparse_emb list\n",
        "    return sparse_emb"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PIFAEmBqoygv"
      },
      "source": [
        "Let's write another function to help us generate sparse vectors in batches."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "id": "7zMnixVGjbl8"
      },
      "outputs": [],
      "source": [
        "def generate_sparse_vectors(context_batch):\n",
        "    # create batch of input_ids\n",
        "    inputs = tokenizer(\n",
        "            context_batch, padding=True,\n",
        "            truncation=True,\n",
        "            max_length=512\n",
        "    )['input_ids']\n",
        "    # create sparse dictionaries\n",
        "    sparse_embeds = build_dict(inputs)\n",
        "    return sparse_embeds"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3EXN_DWknlKT"
      },
      "source": [
        "## Dense Vectors"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YA4muHaL_f--"
      },
      "source": [
        "Alongside sparse vectors we must also add dense vectors (as usual). We do this like so:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 604,
          "referenced_widgets": [
            "2f38b3527fe14fb1b9958613d0f7e386",
            "e94509b6f25e4d83a75ac096256a4771",
            "b67a6ddd1758479796a57c8a21217b56",
            "0e2d25ed676c4fd0ad4ea487aa69001a",
            "64572e360eeb40d0b16db9dc1de0a099",
            "b75b11a6c4b64726bac1adf90af73d3a",
            "5b0c9216b5d144be82bbb7630fbb790f",
            "a559d17f5c1344d1a2e5a0f4f6547fd6",
            "3b796a84261c415c84d1457f4cce7844",
            "10f564e660a74046a731742c0a1a8f66",
            "51991656340c433c8d880bb8d448a05d",
            "b8e6aa1194c34d4e968b77932bd85fe2",
            "63e1f1da82ed413ebf57cebe64a2f198",
            "329c4daaa8e2443b893dfe48f2177486",
            "71a04cb9ad9a41cba499c4c83f2b000c",
            "4663c6ec8e72414e993b13267f9459f7",
            "212a288b630a4f8e997b1fef8df1e8a4",
            "f1dfcf865aff49238c8a6486670cef4a",
            "e03737ba0fcd43729b0089876b62df59",
            "f8662e9941424646a1dc6202a5f80724",
            "9d0c1a85a588423daffb6243285df9a8",
            "d8925d7af4b846babef176fdf1a8d52b",
            "150d19b8f75a46bf9033329151300aaa",
            "8844daa615b64d7c9e03fc2b86f86475",
            "f39a73b76254403887d7dd368a94192b",
            "a8f615a97a9243cdb006741b4ef75ee0",
            "5204c1bc87da418aaaccf01bc0af4ebd",
            "f88046bef3a4446eafe177740273dbc9",
            "c7e8934d03214661a27f3fadc6ba11c1",
            "e609014a61d745a387ee987a25c40d69",
            "10949c8beea943ac9b77ab3015f5a87c",
            "68b623c1fe0341388acbcf4941c97ab1",
            "e2ace8eaeff9483282cc1f38328c3991",
            "e73c452e5ce3412eb2904bda839e2651",
            "be0296955d734a279b7fa522ca1113eb",
            "d04e7259844b4cd2ac142141f8fe087a",
            "4e16d098a2bb4ce887d00bb2ea6ff1d2",
            "27eee5b1b7724e88b57e98e065e48396",
            "6365f53b14804a0c9b9a4e665ffc3d37",
            "250d007acfb04880828b637382def739",
            "55e72569dcfd4fa3b837938aaac1830a",
            "879f71fbec3f48a0b691230da8de331d",
            "460075280d6041adba5760d3a196d95f",
            "ac49b19262934afaa22828726ec46163",
            "6f49c63b4e8c456ea497f9a5a7100844",
            "1d6a110022c5417f916e91e88580ea6f",
            "0ca7dc5cf22a4e3e8b2278d016b54cc2",
            "003de4aa90ce4e6a8085e6759c1d3657",
            "668ee231e19b4c159e78b2d594c2cfd7",
            "242bcddc6a4b43108fe3287fa6a74e5b",
            "20066a20d8b342e7a1858c5fc1001a0a",
            "dc3d8127bd964b0785f22a0075c5c669",
            "f667389b1c5a452aa667ff5979f2e558",
            "313e3b15e2554081879b2e0d1e60a00f",
            "1c8a24e9b5c84a95ad8f0ced8c7cc5a9",
            "761c6312871f4e8393f2a7ba2b3c106a",
            "d89b579d0cf34c7da794d82163a205d7",
            "6c739809d3624a85b804812a5f39f3e0",
            "6aaf9fc8a29441e8a5ba280da2979e78",
            "5b4b14f4b7d54b4f97d2a761c47fcec5",
            "9001c40576a84d9fbfb0c0ad5c37d008",
            "4aec5b65862943ad8ca87376dc8cbb2e",
            "78bd055b1caa43f9a24078edbd666dba",
            "b9643cc066c241488382003753373a65",
            "2d53c7d95a214ad8b5498b718678007d",
            "ad18050a5d6540c99d0b4841fb7b9c7d",
            "3a7ba809f90b4a0d9faf6f3345a5b995",
            "f4a6f0a3fec547809078882a388a6395",
            "8dbd664e9cbb495293d56ba9f0d84ca8",
            "cc39baaa482244d9ad3c84593b92107c",
            "8eca35eb6c724b6899f63ccddbc3d8c5",
            "cbfe212bb58d4c3cb6b3e6e5009a7ef1",
            "48320376c7fa42b59d6e13a1d07a0a4e",
            "fe8746e5d43640d8af51f9de15e5e933",
            "3929f53771614731aee73db0b8edd407",
            "3319b49a33b24645a43f485ad4d758f7",
            "35f36fe3c3b9481c82234cbceea97834",
            "9458ef6a56734af9bea9a44d4f7a2e2a",
            "19ac7eff95df48e088a0ee182ab794da",
            "7587f836d2e640778ec0323077192e50",
            "85731d5f91be42d0a3410b23112f6c65",
            "15ed4d9baae84df892593d8f804b01c0",
            "68c4542e6d374d9bbb8c341da76c7dbe",
            "26a4393c3efc427493592a89553d1513",
            "9e83b7bec0d046c58b22b92fc9ac5ae1",
            "6f33a7846815400494b52cb2ccfd06ca",
            "b1398b1f5cfe476fb4db73c4692c057c",
            "5168bb11858b472c9d9e5b66d8871e4e",
            "6d895fc0eb5842a69dc226f2334c1093",
            "b86e4c2ba33245869b70880c1f51b944",
            "50d2a1b24ad541e4b0b7a9637305d124",
            "8afdfa9a2c68408db2d10386bee09e13",
            "6a88c29c405f4dd1b74b379d5b2c6e55",
            "f2ba281b62444ee28366aa400c7ec515",
            "439acdf13f5d4d438807f666bca32a2e",
            "cb2f0b9e50a94c389cea20870bf9e5b4",
            "0c02920c298244c5bf4533768cde34d6",
            "c89d19150cac4b3a8ed9a7f1f78adcd8",
            "2066514c18fb435683d60b15c1d226be",
            "e983e10ade834d7ba1a7cef977799788",
            "75ad4c8f201b40fa868d7d54f5ad5fd5",
            "79c39c31f90049b888fd0f03b583557c",
            "c248312edbb54c6c8bf9e05ddd77ca5e",
            "b892fb412c4241df99965abb5328ce54",
            "99d5d24ec0224cfbaf63edabc8476a22",
            "ea1435593e224ed8839000ba07d26f36",
            "2e3b4bb7f5d14e4191378693eceddbb5",
            "766f5a4fb1c64932b326091464836a88",
            "986922bd286a4c0ab7eb46da62dafebe",
            "2aab345be9e8455091deb5b9bd349315",
            "460b7881acae4814a331a78a5338188f",
            "51cad4cee9304d878d312b5a3610e6bf",
            "075f3e56475e471fb8c79ee7de01cddd",
            "b4e726c7204d423cbb49a01e24b68a8d",
            "4053d7505b5443cc87432c72d356896d",
            "291244b351dd4b2b9989ce07d12af334",
            "c50df4b1f4e74b2395bc78acbc4b0aed",
            "eca3af151eb14a65b2ceecfd608270da",
            "d7fe3fa53b754766b09b2649964d30f3",
            "5505b0fc5aa14d2497fc34208f226f46",
            "5e020b125d5f4a488353e9f8ef16f7df",
            "ca8035ce703843c083226b84a5b2d2b2",
            "090c7e0b66fc454d886e3ed16e91a5f2",
            "859afece210c4ec6a4a84abd093c79c2",
            "9516bd6ded7d449da93432cbcc010756",
            "60db9656a68546d0b89ff40c2503d469",
            "06d3f29c28f540958c0e31b4f8e97d19",
            "4c49091821d94347bcf836d8adbddace",
            "2dbde5c8886443fc812cc817b16f671d",
            "023f2ceb59444deab0d5a05738f2cdf1",
            "3b531535c75d403aba7ee4721834db99",
            "3ddbda05a3c848029f8fa1be8d19dfd7",
            "03052312477a4ca1b67171e42d085a55",
            "48f977ae07d848f38b0025c84d6a0699",
            "4df74cee7b3a4759aa2a6387857bed51",
            "585fe2b8d21f4326b0c940ba3bd7351d",
            "5e4bfe20f3724d9cbb1e8f8b6dd89953",
            "6d4d03cbcf8b410d98360d34d4d32f4e",
            "565d3df4e92046fca4e8aada4c859931",
            "5491f04a28664bed8be38a5494b7ccd1",
            "44a3bee7d17f42548ed588b9d2a5af4f",
            "09405ad7aece4a78803ec67ed94ee23e",
            "3492fc314f744fe0a07f193a215c1bc8",
            "45074d003c0a4c1eb2b8b10c4252aa9f",
            "57eef46cd24f430da5fb38f03c435ea7",
            "2a162fb5861e495194f47eef8edcd49d",
            "1fc15a1b4e9848db900e70f733037afe",
            "ef2464eee836437e8b0a44f8715fec69",
            "12d7bea2e364417ab82a4940489ca3a2",
            "00057684117d455884b33ae0219a30b5",
            "3c6d6d178a1a47b09af62a461023a5d5",
            "c3b00d42d9e94805bc81838396c618fa",
            "0d23948a762a4d33b86fa459d8685511",
            "745f28a66948420f8d4be5c5e4b995c8"
          ]
        },
        "id": "7_DTvjGw_f-_",
        "outputId": "09a716fb-ad19-460f-92ac-4b882691b91f"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "2f38b3527fe14fb1b9958613d0f7e386",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/737 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "b8e6aa1194c34d4e968b77932bd85fe2",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/190 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "150d19b8f75a46bf9033329151300aaa",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/11.5k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "e73c452e5ce3412eb2904bda839e2651",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/612 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "6f49c63b4e8c456ea497f9a5a7100844",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/116 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "761c6312871f4e8393f2a7ba2b3c106a",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/25.5k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "3a7ba809f90b4a0d9faf6f3345a5b995",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/90.9M [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "9458ef6a56734af9bea9a44d4f7a2e2a",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/53.0 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "6d895fc0eb5842a69dc226f2334c1093",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/112 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "e983e10ade834d7ba1a7cef977799788",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/466k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "460b7881acae4814a331a78a5338188f",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/383 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "ca8035ce703843c083226b84a5b2d2b2",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/13.8k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "03052312477a4ca1b67171e42d085a55",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/232k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "45074d003c0a4c1eb2b8b10c4252aa9f",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/349 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/plain": [
              "SentenceTransformer(\n",
              "  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel \n",
              "  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False})\n",
              "  (2): Normalize()\n",
              ")"
            ]
          },
          "execution_count": 18,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "from sentence_transformers import SentenceTransformer\n",
        "\n",
        "# load a sentence transformer model from huggingface\n",
        "model = SentenceTransformer(\n",
        "    'multi-qa-MiniLM-L6-cos-v1',\n",
        "    device='cuda'\n",
        ")\n",
        "model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mkdXK8b4_f-_"
      },
      "source": [
        "The model gives us a `384` dimensional dense vector."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wph64ujXrhrp"
      },
      "source": [
        "## Upsert Documents"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oX99CnOroEq0"
      },
      "source": [
        "Now we can go ahead and generate sparse and dense vectors for the full dataset and upsert them along with the metadata to the new hybrid index. We can do that easily as follows:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 118,
          "referenced_widgets": [
            "7a01cfb8ac7c4d2293efca7f8030f407",
            "550a296587d24a3f867f45b09273648c",
            "b8e4d284fc124a6eb75999df4bff1bc5",
            "f396028be4d845349e542efde6fee7f2",
            "1f173bb2a3cf4b3396124f62a0964d85",
            "7f33ae1dce5241b9b4aa19368a00de3a",
            "1e7c98d2ac40410d84514111fe61dcc1",
            "3964c2274b6746f8aa0a79ec8a36c338",
            "6c5e8672edab4fff968943bdec9154fc",
            "d98293d4f95042f69a4db00fcd3b60fe",
            "172a255a494e4eb8a012d49c3259c21c"
          ]
        },
        "id": "vcM5h0T1_f-_",
        "outputId": "d6a55761-a405-4bde-b949-fb4cdb59ab05"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "7a01cfb8ac7c4d2293efca7f8030f407",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "  0%|          | 0/32 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/plain": [
              "{'namespaces': {'': {'vectorCount': 1000}},\n",
              " 'dimension': 384,\n",
              " 'indexFullness': 0,\n",
              " 'totalVectorCount': 1000}"
            ]
          },
          "execution_count": 19,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "from tqdm.auto import tqdm\n",
        "\n",
        "batch_size = 32\n",
        "\n",
        "for i in tqdm(range(0, len(contexts), batch_size)):\n",
        "    # find end of batch\n",
        "    i_end = min(i+batch_size, len(contexts))\n",
        "    # extract batch\n",
        "    context_batch = contexts[i:i_end]\n",
        "    # create unique IDs\n",
        "    ids = [str(x) for x in range(i, i_end)]\n",
        "    # add context passages as metadata\n",
        "    meta = [{'context': context} for context in context_batch]\n",
        "    # create dense vectors\n",
        "    dense_embeds = model.encode(context_batch).tolist()\n",
        "    # create sparse vectors\n",
        "    sparse_embeds = generate_sparse_vectors(context_batch)\n",
        "\n",
        "    vectors = []\n",
        "    # loop through the data and create dictionaries for uploading documents to pinecone index\n",
        "    for _id, sparse, dense, metadata in zip(ids, sparse_embeds, dense_embeds, meta):\n",
        "        vectors.append({\n",
        "            'id': _id,\n",
        "            'sparse_values': sparse,\n",
        "            'values': dense,\n",
        "            'metadata': metadata\n",
        "        })\n",
        "\n",
        "    # upload the documents to the new hybrid index\n",
        "    index.upsert(vectors=vectors)\n",
        "\n",
        "# show index description after uploading the documents\n",
        "index.describe_index_stats()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ir1n8ABfqbV-"
      },
      "source": [
        "## Querying"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vv24_cjVrVKz"
      },
      "source": [
        "Now we can query the index, providing the sparse and dense vectors of a question, along with a weight for keyword relevance (\u201calpha\u201d). `Alpha=1` will provide a purely semantic-based search result and `alpha=0` will provide a purely keyword-based result equivalent to BM25. The default value is `0.5`."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IedLiNIEqdWi"
      },
      "source": [
        "Let's write a helper function to execute queries and after that run some queries."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "id": "HuIoyi6Trwqc"
      },
      "outputs": [],
      "source": [
        "from pinecone_text.hybrid import hybrid_convex_scale\n",
        "\n",
        "def hybrid_query(question, top_k, alpha):\n",
        "    # convert the question into a sparse vector\n",
        "    sparse_vec = generate_sparse_vectors([question])\n",
        "    # convert the question into a dense vector\n",
        "    dense_vec = model.encode([question]).tolist()\n",
        "    dense_vec, sparse_vector = hybrid_convex_scale(\n",
        "        dense_vec, sparse_vec, alpha=alpha\n",
        "    )\n",
        "    # query pinecone with the query parameters\n",
        "    result = index.query(\n",
        "        vector=dense_vec,\n",
        "        sparse_vector=sparse_vec[0],\n",
        "        top_k=top_k,\n",
        "        include_metadata=True,\n",
        "      )\n",
        "    # return search results as json\n",
        "    return result"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "id": "iszWr04qvONn"
      },
      "outputs": [],
      "source": [
        "question = \"Can clinicians use the PHQ-9 to assess depression in people with vision loss?\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TBjad1-cvRlA"
      },
      "source": [
        "First, we will do a pure semantic search by setting the alpha value as 1."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jQkFj1b_qjfx",
        "outputId": "fc876ac3-347c-46c9-b779-f1117f85f4db"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'matches': [{'id': '305',\n",
              "   'score': 0.732677519,\n",
              "   'values': [],\n",
              "   'sparseValues': {},\n",
              "   'metadata': {'context': 'The gap between evidence-based treatments and routine care has been well established. Findings from the Sequenced Treatments Alternatives to Relieve Depression (STAR*D) emphasized the importance of measurement-based care for the treatment of depression as a key ingredient for achieving response and remission; yet measurement-based care approaches are not commonly used in clinical practice.\\nThe Nine-Item Patient Health Questionnaire (PHQ-9) for monitoring depression severity was introduced in 19 diverse psychiatric practices. During the one-year course of the project the helpfulness and feasibility of implementation of PHQ-9 in these psychiatric practices were studied. The project was modeled after the Institute for Healthcare Improvement Breakthrough Series. Two of the 19 practices dropped out during the course of the project.\\nBy the conclusion of the study, all remaining 17 practices had adopted PHQ-9 as a routine part of depression care in their practice. On the basis of responses from 17 psychiatrists from those practices, PHQ-9 scores influenced clinical decision making for 93% of 6,096 patient contacts. With the additional information gained from the PHQ-9 score, one or more treatment changes occurred during 40% of these clinical contacts. Changing the dosage of antidepressant medication and adding another medication were the most common treatment changes recorded by psychiatrists, followed by starting or increasing psychotherapy and by switching or initiating antidepressants. In 3% of the patient contacts, using the PHQ-9 led to additional suicide risk assessment.'}},\n",
              "  {'id': '711',\n",
              "   'score': 0.696874201,\n",
              "   'values': [],\n",
              "   'sparseValues': {},\n",
              "   'metadata': {'context': \"To investigate whether the Patient Health Questionnaire-9 (PHQ-9) possesses the essential psychometric characteristics to measure depressive symptoms in people with visual impairment.\\nThe PHQ-9 scale was completed by 103 participants with low vision. These data were then assessed for fit to the Rasch model.\\nThe participants' mean +/- standard deviation (SD) age was 74.7 +/- 12.2 years. Almost one half of them (n = 46; 44.7%) were considered to have severe vision impairment (presenting visual acuity<6/60 in the better eye). Disordered thresholds were evident initially. Collapsing the two middle categories produced ordered thresholds and fit to the Rasch model (chi = 10.1; degrees of freedom = 9; p = 0.34). The mean (SD) items and persons Fit Residual values were -0.31 (1.12) and -0.25 (0.78), respectively, where optimal fit of data to the Rasch model would have a mean = 0 and SD = 1. Unidimensionality was demonstrated confirming the construct validity of the PHQ-9 and there was no evidence of differential item functioning on a number of factors including visual disability. The person separation reliability value was 0.80 indicating that the PHQ-9 has satisfactory precision. There was a degree of mistargeting as expected in this largely non-clinically depressed sample.\"}},\n",
              "  {'id': '735',\n",
              "   'score': 0.505706906,\n",
              "   'values': [],\n",
              "   'sparseValues': {},\n",
              "   'metadata': {'context': \"To assess whether perspective-taking, which researchers in other fields have shown to induce empathy, improves patient satisfaction in encounters between student-clinicians and standardized patients (SPs).\\nIn three studies, randomly assigned students (N = 608) received either a perspective-taking instruction or a neutral instruction prior to a clinical skills examination in 2006-2007. SP satisfaction was the main outcome in all three studies. Study 1 involved 245 third-year medical students from two universities. Studies 2 and 3 extended Study 1 to examine generalizability across student and SP subpopulations. Study 2 (105 physician assistant students, one university) explored the effect of perspective-taking on African American SPs' satisfaction. Study 3 (258 third-year medical students, two universities) examined the intervention's effect on students with high and low baseline perspective-taking tendencies.\\nIntervention students outscored controls in patient satisfaction in all studies: Study 1: P = .01, standardized effect size = 0.16; Study 2: P = .001, standardized effect size = 0.31; Study 3: P = .009, standardized effect size = 0.13. In Study 2, perspective-taking improved African American SPs' satisfaction. In Study 3, intervention students with high baseline perspective-taking tendencies outscored controls (P = .0004, standardized effect size = 0.25), whereas those with low perspective-taking tendencies did not (P = .72, standardized effect size = 0.00).\"}}],\n",
              " 'namespace': ''}"
            ]
          },
          "execution_count": 22,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "hybrid_query(question, top_k=3, alpha=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "88o5Cn-FwF-1"
      },
      "source": [
        "The most relevant result from above is the second document with id 711. Now let's try with an alpha value of 0.3."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HjXgq7OQwmfM",
        "outputId": "daee78ed-3b7e-4c52-f95f-522408c05dc9"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'matches': [{'id': '711',\n",
              "   'score': 0.486920714,\n",
              "   'values': [],\n",
              "   'sparseValues': {},\n",
              "   'metadata': {'context': \"To investigate whether the Patient Health Questionnaire-9 (PHQ-9) possesses the essential psychometric characteristics to measure depressive symptoms in people with visual impairment.\\nThe PHQ-9 scale was completed by 103 participants with low vision. These data were then assessed for fit to the Rasch model.\\nThe participants' mean +/- standard deviation (SD) age was 74.7 +/- 12.2 years. Almost one half of them (n = 46; 44.7%) were considered to have severe vision impairment (presenting visual acuity<6/60 in the better eye). Disordered thresholds were evident initially. Collapsing the two middle categories produced ordered thresholds and fit to the Rasch model (chi = 10.1; degrees of freedom = 9; p = 0.34). The mean (SD) items and persons Fit Residual values were -0.31 (1.12) and -0.25 (0.78), respectively, where optimal fit of data to the Rasch model would have a mean = 0 and SD = 1. Unidimensionality was demonstrated confirming the construct validity of the PHQ-9 and there was no evidence of differential item functioning on a number of factors including visual disability. The person separation reliability value was 0.80 indicating that the PHQ-9 has satisfactory precision. There was a degree of mistargeting as expected in this largely non-clinically depressed sample.\"}},\n",
              "  {'id': '305',\n",
              "   'score': 0.461695194,\n",
              "   'values': [],\n",
              "   'sparseValues': {},\n",
              "   'metadata': {'context': 'The gap between evidence-based treatments and routine care has been well established. Findings from the Sequenced Treatments Alternatives to Relieve Depression (STAR*D) emphasized the importance of measurement-based care for the treatment of depression as a key ingredient for achieving response and remission; yet measurement-based care approaches are not commonly used in clinical practice.\\nThe Nine-Item Patient Health Questionnaire (PHQ-9) for monitoring depression severity was introduced in 19 diverse psychiatric practices. During the one-year course of the project the helpfulness and feasibility of implementation of PHQ-9 in these psychiatric practices were studied. The project was modeled after the Institute for Healthcare Improvement Breakthrough Series. Two of the 19 practices dropped out during the course of the project.\\nBy the conclusion of the study, all remaining 17 practices had adopted PHQ-9 as a routine part of depression care in their practice. On the basis of responses from 17 psychiatrists from those practices, PHQ-9 scores influenced clinical decision making for 93% of 6,096 patient contacts. With the additional information gained from the PHQ-9 score, one or more treatment changes occurred during 40% of these clinical contacts. Changing the dosage of antidepressant medication and adding another medication were the most common treatment changes recorded by psychiatrists, followed by starting or increasing psychotherapy and by switching or initiating antidepressants. In 3% of the patient contacts, using the PHQ-9 led to additional suicide risk assessment.'}},\n",
              "  {'id': '130',\n",
              "   'score': 0.383923233,\n",
              "   'values': [],\n",
              "   'sparseValues': {},\n",
              "   'metadata': {'context': \"Although record linkage of routinely collected health datasets is a valuable research resource, most datasets are established for administrative purposes and not for health outcomes research. In order for meaningful results to be extrapolated to specific populations, the limitations of the data and linkage methodology need to be investigated and clarified. It is the objective of this study to investigate the differences in ascertainment which may arise between a hospital admission dataset and a dispensing claims dataset, using major depression in pregnancy as an example. The safe use of antidepressants in pregnancy is an ongoing issue for clinicians with around 10% of pregnant women suffer from depression. As the birth admission will be the first admission to hospital during their pregnancy for most women, their use of antidepressants, or their depressive condition, may not be revealed to the attending hospital clinicians. This may result in adverse outcomes for the mother and infant.\\nPopulation-based de-identified data were provided from the Western Australian Data Linkage System linking the administrative health records of women with a delivery to related records from the Midwives' Notification System, the Hospital Morbidity Data System and the national Pharmaceutical Benefits Scheme dataset. The women with depression during their pregnancy were ascertained in two ways: women with dispensing records relating to dispensed antidepressant medicines with an WHO ATC code to the 3rd level, pharmacological subgroup, 'N06A Antidepressants'; and, women with any hospital admission during pregnancy, including the birth admission, if a comorbidity was recorded relating to depression.\\nFrom 2002 to 2005, there were 96698 births in WA. At least one antidepressant was dispensed to 4485 (4.6%) pregnant women. There were 3010 (3.1%) women with a comorbidity related to depression recorded on their delivery admission, or other admission to hospital during pregnancy. There were a total of 7495 pregnancies identified by either set of records. Using data linkage, we determined that these records represented 6596 individual pregnancies. Only 899 pregnancies were found in both groups (13.6% of all cases). 80% of women dispensed an antidepressant did not have depression recorded as a comorbidity on their hospital records. A simple capture-recapture calculation suggests the prevalence of depression in this population of pregnant women to be around 16%.\"}}],\n",
              " 'namespace': ''}"
            ]
          },
          "execution_count": 23,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "hybrid_query(question, top_k=3, alpha=0.3)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eqZ93O95w2oe"
      },
      "source": [
        "The most relevant document is now ranked the highest."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1YY3hiL3xNQ5"
      },
      "source": [
        "# Delete the Index"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DnGtG5iaUArB"
      },
      "outputs": [],
      "source": [
        "pc.delete_index(\"hybrid-test\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jD3z5NI7svcZ"
      },
      "source": [
        "---"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [],
      "provenance": []
    },
    "environment": {
      "kernel": "python3",
      "name": "common-cu110.m95",
      "type": "gcloud",
      "uri": "gcr.io/deeplearning-platform-release/base-cu110:m95"
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.12 (main, Apr  5 2022, 01:52:34) \n[Clang 12.0.0 ]"
    },
    "vscode": {
      "interpreter": {
        "hash": "b8e7999f96e1b425e2d542f21b571f5a4be3e97158b0b46ea1b2500df63956ce"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}